Papers with multi-modal representation learning
Domain-aware Self-supervised Pre-training for Label-Efficient Meme Analysis (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing self-supervised learning strategies focus on uni-modal applications . a recent study shows that multimodality is a major challenge for multi-modal systems . |
| Approach: | They propose two self-supervised pre-training methods that employ off-the-shelf multi-modal hate-speech data . they also incorporate multiple specialized pretext tasks to cater to complex multi-modity representation learning . |
| Outcome: | The proposed methods outperform the baseline self-supervised learning strategies on the Memotion challenge and the HarMeme task. |
Parallel Attention Network with Sequence Matching for Video Grounding (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to video grounding are sensitive to quality of proposals and inefficient because all proposal-query pairs are compared. |
| Approach: | They propose a Parallel Attention Network with Sequence matching to capture selfmodal contexts and cross-modal attentive information between video and text. |
| Outcome: | The proposed approach is superior to state-of-the-art methods on three datasets. |
A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing multi-modal neural machine translation models do not fully exploit fine-grained semantic correspondences between semantic units of different modalities. |
| Approach: | They propose a graph-based multi-modal fusion encoder that exploits fine-grained semantic correspondences between different modalities. |
| Outcome: | The proposed encoder significantly extends the conventional text-based translation by taking images as additional inputs. |
Soft-Prompting with Graph-of-Thought for Multi-modal Representation Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to learn multi-modal tasks are based on chain-of-thought . however, human thought processes are non-linear and employ dynamic adjustment and updating mechanisms. |
| Approach: | They propose a chain-of-thought technique that adjusts the length of the chain to improve the performance of generated prompts. |
| Outcome: | The proposed model improves multi-modal representation learning in visual, visual, and audio-visual tasks and also has good domain generalization performance due to better reasoning. |